Electro-optic vision systems

ABSTRACT

An image processing system for delivering real scene information to a data processor. The system includes the data processor, an image-delivery mechanism, an information delivery mechanism, and a graphic processor.

CROSS-REFERENCES TO RELATED PATENT APPLICATIONS

This application is a continuation-in-part of the following patentapplications: Ser. No. 08/691,784, filed Aug. 2, 1996; Ser. No.08/119,360, filed on Sep. 10, 1993; Ser. No. 08/307,360, filed on Sep.14, 1994; Ser. No. 08/335,912, filed on Nov. 8, 1994, Ser. No.08/335,940, filed on Nov. 8, 1994; Ser. No. 08/335,710, filed on Dec. 4,1994; Ser. No. 08/441,299, filed on Mar. 27, 1995; Ser. No. 08/480,689,filed on Jun. 7, 1995; Ser. No. 08/482,943, filed on Jun. 7, 1995; andSer. No. 08/571,096, filed on Dec. 12, 1995.

FIELD OF THE INVENTION

The present invention is generally related to electronic vision devicesand methods, and is specifically related to image augmentation incombination with navigation, position, and attitude devices.

BACKGROUND OF THE INVENTION

One may have to look quite far into the annals of history to find thefirst uses of maps. Maps generally provide information to alert a userto things that are not readily apparent from simple viewing of a realscene from the users location. For example, a user of a city road mapmay not be able to see a tunnel on Elm Street if the user is currentlyseven miles away on First Street and looking in the direction of the ElmStreet tunnel. However, from the First Street location, the user coulddetermine from a road map that there is a tunnel on Elm Street. He couldlearn that the tunnel is three miles long, starts on Eighth Street andends on Eleventh Street. There may even be an indication of the size ofthe tunnel such that it could accommodate four traffic lanes and abicycle lane.

Unfortunately, it is not always possible to translate the informationfrom a map to the real scene that the information represents as thescene is actually viewed. It is common for users of maps to attempt toalign the map to reality to get a better “feel” of where things are inrelation to the real world. Those who are familiar with maps can verifythat the fact that maps are drawn with north being generally in thedirection of the top of the map, is of little use when translating theinformation to the scene of interest. Regardless of where north is, onetends to turn the map so that the direction ahead of the user, or in thedirection of travel, in a real scene matches that direction on the map.This may result in the condition of an “upside down” map that is quitedifficult to read (the case when the user is traveling south). Althoughtranslating the directions of the map to reality is a formidable task,it is an even greater problem to translate the symbols on the map tothose objects in reality which they represent. The tunnel symbol on themap does not show what the real tunnel actually looks like. The factthat the appearance of the tunnel from infinitely many points of view isprohibitively difficult to represent on a map accounts for the use of asimple symbol. Furthermore, the map does not have any indication fromwhich point of view the user will first see the tunnel, nor anyindication of the path which the user will take to approach the tunnel.

It is now possible to computerize city road map information and displaythe maps according to the path taken by a user. The map is updated in“real-time” according to the progress of the user through the citystreets. It is therefore possible to relieve the problem of upside-downmaps as the computer could re-draw the map with the text in correctorientation relative to the user even when one is traveling in asoutherly direction. The computer generated map is displayed at amonitor that can be easily refreshed with new information as the userprogresses along his journey. Maps of this type for automobiles are wellknown in the art. Even very sophisticated maps with computer generatedindicia to assist the user in decision making are available anddescribed in patents such as DeJong U.S. Pat. No. 5,115,398. This devicecan display a local scene as it may appear and superimpose onto thescene, symbolic information that suggests an action to be taken by theuser. Even in these advanced systems, a high level of translation isrequired of the user. The computer generated map does not attempt topresent an accurate alignment of displayed images to the real objectwhich they represent.

Devices employing image supplementation are known and include Head UpDisplays (HUDs) and Helmet Mounted Displays (HMDs). A HUD is a usefulvision system which allows a user to view a real scene, usually throughan optical image combiner such as a holographic mirror or a dichroicbeamsplitter, and have superimposed thereon, navigational informationfor example symbols of real or imaginary objects, vehicle speed andaltitude data, et cetera. It is a primary goal of the HUD to maximizethe time that the user is looking into the scene of interest. For afighter pilot, looking at a display device located nearby on aninstrument panel, and changing the focus of ones' eyes to read thatdevice, and to return to the scene of interest, requires a criticallylong time and could cause a fatal error. A HUD allows a fighter pilot tomaintain continuous concentration on a scene at optical infinity whilereading instruments that appear to the eye to also be located at opticalinfinity and thereby eliminating the need to refocus ones' eyes. A HUDallows a pilot to maintain a “head-up” position at all times. For theairline industry, HUDs have been used to land airplanes in lowvisibility conditions. HUDs are particularly useful in a landingsituation where the boundaries of a runway are obscured in the pilotsfield of view by fog but artificial boundaries can be projected onto theoptical combiner of the HUD system to show where in the user's visionfield the real runway boundaries are. The virtual runway projection ispositioned in the vision field according to data generated bycommunication between a computer with and the airport instrument landingsystem (ILS) which employs a VHF radio beam. The system provides thecomputer with two data figures. First a glide slope figure, and second,a localizer which is a lateral position figure. With these data, thecomputer is able to generate an optical image (photon) to be projectedand combined with the real scene (photon) that passes through thecombiner and thereby enhancing certain features of the real scene; forexample runway boundaries. The positioning of the overlay depends on theaccuracy of the airplane boresight being in alignment with the ILS beamand other physical limitations. The computer is not able to recognizeimages in the real scene and does not attempt to manipulate the realscene except for highlighting parts thereof. HUDs are particularlycharacterized in that they are an optical combination of two photonscenes. The combination being a first scene, one that is normally viewedby the users eyes passes through an optical combiner, and a second,computer generated photon image which is combined with the real image atan optical element. In a HUD device it is not possible for the computerto address objects of the real scene, for example to alter or deletethem. The system only adds enhancement to a feature of the real image bydrawing interesting features thereon. Finally, HUDs are very bulky andare typically mounted into an airplane or automobile and require a greatdeal of space and complex optics including holograms and speciallydesigned lenses.

HMDs are similar to HUDs in that they also combine enhancement imageswith real scene photon images but they typically have very portablecomponents. Micro CRTs and small combiners make the entire system helmetmountable. It is a complicated matter to align computer generated imagesto a real scene in relation to a fast moving helmet. HUDs can align thedata generated image that is indexed to the slow moving airplane axiswhich moves slowly in relation to a runway. For this reason, HMDsgenerally display data that does not change with the pilots headmovements such as altitude and airspeed. HMDs suffer the same limitationas the HUDs in that they do not provide the capacity to remove oraugment elements of the real image.

Another related concept that has resulted in a rapidly developing fieldof computer assisted vision systems is known as virtual reality (VR).Probably best embodied in the fictional television program “Star Trek;The Next Generation”, the “Holodeck” is a place where a user can go tohave all of his surroundings generated by a computer so as to appear tothe user to be another place or another place and time.

Virtual reality systems are useful in particular for a training means.For example in aircraft simulation devices. A student pilot can besurrounded by a virtual “cockpit” which is essentially a computerinterface whereby the user “feels” the environment that may be presentin a real aircraft, in a very real way and perhaps enhanced withcomputer generated sounds, images and even mechanical stimuli. Actionstaken by the user may be interpreted by the computer and the computercan respond to those actions to control the stimuli that surround theuser. VR machines can create an entire visual scene and there is noeffort to superimpose a computer generated scene onto a real scene. A VRdevice generally does not have any communication between its actuallocation in reality and the stimuli being presented to the user. Thelocation of the VR machine and the location of the scene being generatedgenerally have no physical relationship.

VR systems can be used to visualize things that do not yet exist. Forexample, a home can be completely modeled with a computer so that apotential buyer can “walk-through” before it is even built. The buyercould enter the VR atmosphere and proceed through computer generatedimages and stimuli that accurately represent what a home would be likeonce it is built. In this way, one could know if a particular style ofhome is likable before the large cost of building the home is incurred.The VR machine being entirely programmed with information from adesigner does not anticipate things that presently exist and there is nocommunication between the elements presented in the VR system to thoseelements existing in reality.

While the systems and inventions of the prior art are designed toachieve particular goals, features, advantages, and objectives, some ofthose being no less than remarkable, these systems and inventions havelimitations and faults that prevent their use in ways that are onlypossible by way of the present invention. The prior art systems andinventions can not be used to realize the advantages and objectives ofthe present invention.

SUMMARY OF THE INVENTION

The present invention involves a vision system including devices andmethods of augmented reality wherein an image of some real scene isaltered by a computer processor to include information from a data basehaving stored information of that scene in a storage location that isidentified by the real time position and attitude of the vision system.

One embodiment of the present invention comprises an image processingsystem used in an electro-optic apparatus having an image capturingmeans, position determining means, attitude determining means, databaseof real scene information, and a display means. The image capturingmeans generating a digital image representing an object. The imageprocessing system comprises a data processor and a graphic processorgenerating an image to be displayed by the display means based on datagenerated by the data processor. The image processing system furthercomprises means for delivering the digital image to the data processorand means for identifying information related to the digital image fromthe database information. The related information is delivered to thedata processor. The data processor combining and processing the digitalimage and the related information. As a result, a user of theelectro-optic apparatus can see an augmented image

These and other features of the present invention will become apparentfrom the following description when read in conjunction with thedrawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an electro-optic system of the presentinvention.

FIG. 2 is a block diagram showing an image processing system of thepresent invention used in the electro-optic system of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

The present invention comprises a novel electro-optic system andassociated image processing devices. The following description ispresented to enable any person skilled in the art to make and use theinvention. Descriptions of specific applications are provided only asexamples. Various modifications to the preferred embodiments will bereadily apparent to those skilled in the art, and the general principlesdefined herein may be applied to other embodiments and applicationswithout departing from the spirit and scope of the invention. Thus, thepresent invention is not intended to be limited to the embodimentsshown, but is to be accorded the widest scope consistent with theprinciples and features disclosed herein.

FIG. 1 is a block diagram of an electro-optic system 10 of the presentinvention. System 10 comprises a position determining means 16, anattitude determining means 15, a camera 19, an image processing unit 14,and a display 13. Camera 19 comprises an electro-optic device capable ofconverting a photon input (from a field of view) into an electronicimage. Camera 19 then transmits the electronic image to image processingunit 14. Position determining means 16 determines the position of thecamera and transmit the information to image processing unit 14.Similarly, attitude determining means determines the camera pointingattitude of the camera defined by, e.g., the symmetry axis of the camerafield of view. Attitude determining means 15 transmits the attitudeinformation to image processing unit 14. Image processing unit 14analyzes and processes inputs from position determining means 16,attitude determining means 15, and camera 19 so as to generate real andaugmented images. These images are then transmitted to display 13 foruse by the user. Various applications and advantages of electro-opticsystem 10 have been disclosed in the following copending patentapplications: Ser. No. 08/119,360, entitled “An Electro-Optic VisionSystem Which Exploits Position and Attitude” and filed Sep. 10, 1993;Ser. No. 08/307,360, entitled “Vision System For Viewing A SportingEvent” and filed Sep. 14, 1994; Ser. No. 08/335,912, entitled “VisionImaging Devices and Methods Exploiting Position And Attitude” and filedNov. 8, 1994; Ser. No. 08/335,940, entitled “Vision Imaging Devices AndMethods Having An Unlimited Zoom Range” and filed Nov. 8, 1994; Ser. No.08/335,710, entitled “Computer Games Having Optically Acquired ImagesWhich Are Combined With Computer Generated Graphics And Images” andfiled Dec. 4, 1994; Ser. No. 08/441,299, entitled “Augmented RealityVision Systems Which Derive Image Information From Other Vision Systems”and filed Mar. 27, 1995; Ser. No. 08/480,689, entitled “Vision SystemsFor Viewing Objects That Identify Themselves” and filed Jun. 7, 1995;Ser. No. 08/482,943, entitled “An Electro-Optic Vision System WhichExploits Position And Altitude” and filed Jun. 7, 1995; Ser. No.08/482,944, entitled “Vision System Computer Modeling Apparatus” andfiled Jun. 7, 1995; Ser. No. 08/571,096, entitled “Computer VisionSystem Which Determines Identify, Position and Orientation Of Objects InA Scene And Displays Augmented Images Thereof” and filed Dec. 12, 1995.These patent applications are incorporated herein by reference.

It should be appreciated that electro-optic system 10 of FIG. 1 maycontains additional components. For example, it may be desirable toinclude a set of user inputs so that an user can enter data to imageprocessing unit 14. Other measuring devices, such as temperature gauge,accelerometer, range finder, etc., can also be coupled to imageprocessing unit 14.

FIG. 2 is a block diagram of an embodiment of an electro-optic system100 showing the detail structure of its image processing unit.

System 100 comprises a video input device, such as a camera 102, whichaccepts optical image information and generates a corresponding digitalvideo in the form of IEEE 1394 (FireWire) data. Camera 102 canoptionally generate information relating to iris, exposure time. zoom(image magnification) ratio and stabilizer data (e.g., 1, 2, or 3 axisacceleration). FireWire is a high speed serial interface busspecification especially tuned for transferring digital videoinformation at rates of up to 400 Megabits per second (Mbits/second).This interface has been implemented in Sony Corporation's DigitalHandyCam series of camcorders. An exemplary camera contains one or morecharged coupled device (CCD) area imaging sensors. It producesinformation which is digitized and encoded to an industry standard videoformat. This data is then transmitted serially at up to 400 Mbits/secondvia the FireWire data bus. Control and configuration information can bepassed bi-directionally over this bus as well.

It should be noted that although IEEE 1394 is the presently preferreddigital interface for video data, the invention could be implementedusing other digital interfaces, now available or to be developed in thefuture.

The output of camera 102 is connected to a FireWire video interfacechipset 104. This chipset accepts the IEEE 1394 data from camera 102 andgenerates command and data in a Zoom Video bus 106 and a PeripheralComponent Interconnect (PCI) bus 108. Zoom Video (or Zoomed Video) is aninterface standard used by Personal Computer Manufacturing Card IndustryAssociation (PCMCIA) card manufacturers and graphics controllermanufacturers to provide a central processing unit (CPU) independentpath to a graphics controller for digital video information. It is a bitparallel (YUV encoded) serial data interface (i.e., the pixels arriveserially for each line). Vertical, horizontal sync as well as dotclockis provided. PCI bus 108 is a 32 (expandable to 64)-bit high speedparallel databus standard. The bus operates currently at 33 Mhz so as toprovide up to 132 Mbytes/second data transfer rate. Futureimplementations will increase the bus speed. This bus is described indetail in the “PCI Specification, version 2.1” published by the PCISpecial Interest Group. Chipset 104 receives the IEEE 1394 serial datastream (at up to 400 Mbits/second) and converts the data into paralleldata suitable for sending over the PCI and/or Zoom Video data busses.The data that goes out on Zoom Video bus 106 is arranged to fit the YUVencoding format with vertical and horizontal syncs. Chipset 104 can alsooperate as a PCI Bus Master, thus can burst image data to anywhere inthe CPU's main memory or to any PCI slave device's memory.

It should again be pointed out that Zoom Video bus 106 and PCI bus 108are exemplary digital buses. The present invention could be implementedusing other high bandwidth buses.

The output of chipset 104 is coupled to an image processor (IP) 110. TheZoom Video bus allows IP 110 to receive video data in parallel with aCPU system, shown in FIG. 2 as reference numeral 114. IP 110 couldperform many tasks, from complex to simple. At the complex end, IP 110may be responsible for processing video on a frame by frame basis toextract data from, or to enhance the image. An example is the V-LACE™real time image enhancement algorithm from DigiVision. At the simpleend, IP 110 may be asked to shut down and passively pass the image datathrough to a 3D Graphics accelerator 116. Other examples may involvefeature extraction from the image, classification of those features andalerting the main CPU of those results.

IP 110 could be implemented as a DSP like subsystem with its own memory,CPU and I/O capability. A high performance parallel execution unit CPUlike the TMS320C80 is preferably used to execute algorithms which mayemploy fast Fourier Transform (FFT) like calculations. An example of IP110 is Ariel Corp's TMS320C80 based Griffin PCI bus image processingboard.

Zoom Video is preferably used to pass the image data in real time to IP110 and deliver the results to 3D graphics processor 116. PCI bus 108provides an alternate path for the result or input data. Main CPU system114 may utilize IP 110 as a parallel processor (to itself),pre-processor, or post-processor of image information.

3D graphics processor 116 is used to off load time-consuming graphicsoperations from the CPU. Although all functions could be implemented inthe main CPU, that would consume a substantial amount of the power ofpresent generation of CPUs, leaving little power for other tasks.Graphics processor 116 receives image information from Zoom Video bus106. It contains a PCI interface, which provides a high bandwidth bus tothe CPU for image rendering primitives, background information, textetc. Graphics processor 116 provides near real time rendering of theaugmentation objects using dedicated hardware specifically designed for3D graphics operations.

Alternately, it is possible for video data to flow at full speed, about27 Mbytes per second, from the FireWire interface over the PCI bus anddirectly to the memory of graphic processor 116.

The image generated by graphics processor 116 is sent to a display 118.It converts RGB encoded digital data into light. The preferred displayis small size, low power consumption, and high resolution. Examples ofsuitable displays are active matrix color liquid crystal display (LCD),LCD projection panel, digital micromirror device (DMD), and vacuumfluorescent display (VFD).

CPU system 114 could be a single CPU. Alternatively, it could be amultiprocessing system. Examples are MIPS 10000, DEC ALPHA, SUN UltraSPARC. The preferred system is a Pentium Pro single or multiprocessorsystem (this choice is based on costs and availability of developmenttools).

The preferred CPU system typically requires a core logic chipset (shownin FIG. 2 as reference numeral 120). It provides the interface betweenCPU system 114, a main memory 122, and PCI data bus 108. Examples ofchipsets are Intel's Orion Core Logic chipset, 440FX, 450GX, and 450KX.The Orion chipset provides multi-processing support for up to fourprocessors. A PCI bus and an Industry Standard Architecture (ISA) bus(shown as reference numeral 126) are supported. The preferred CPU systemalso requires random access memory (RAM) 122 to provide storage forprogram execution.

It should be noted that the use of 3D graphics processor 116 isoptional. Some microprocessors contain multimedia instructions whichallow multimedia tasks to be easily performed. Examples of suchmicroprocessors are Intel's Pentium-MMX (code named the P55C) and Sun'sUltraSPARC. It should also be pointed out that graphics processors couldbe used in combination with this kind of microprocessors (i.e., havinginstructions designed to execute multimedia instructions) in order toobtain enhanced performance.

System 100 comprises a mass storage unit 130, which is coupled to PCIbus 108 by a hard disk interface 131. Examples of interface 131 are anEnhanced Integrated Drive Electronics (EIDE) interface and a SmallComputer System Interface (SCSI). Unit 130 provides storage space forGeographic Information Systems (GIS) database information, algorithm andprogram information for IP 110, and operating system software for mainCPU system 114. System 100 contains software (which could be stored inmass storage unit 130 and loaded into RAM 122 or burnt into read-onlymemory) for searching and retrieving data from the GIS database. Thesearching preferably uses position, attitude, and other data foridentifying the location and point of view of camera 102.

Although there are mass storage units having several gigabytes ofstorage, their physical sizes are too large for the present embodiment.The preferred mass storage unit 130 is a balance between size, weight,cost and performance. At the present time, a rotating magnetic mediumstorage using 1.2 gigabyte 2.5″ technology is considered the preferredstorage unit.

System 100 also comprises a real time clock (RTC) 134. It provides localtimekeeping and (optionally) non-volatile storage of system parametersduring power off conditions. The design of RTC 134 depends on therequirements of CPU system 114 and core chipset 120. Many companies,such as Dallas Semiconductors, Inc., Benchmarq Semiconductors, and Chips& Technologies, Inc. manufacture RTCs for various CPU systemarchitectures.

The connection to various peripheral devices is now described. Serialports (shown as numerals 136, 137 and 160), such as RS232, NMEA-183, andRS422, could be used to provide the connection. The serial ports provideserial to parallel conversion, which converts asynchronously formatteddata (received from the peripheral devices) to parallel data. The datais sent over ISA bus 126 to CPU system 114 for processing. It should benoted that the present invention is not limited to using asynchronousserial ports as means for interfacing with peripheral devices. Forexample, parallel ports or synchronous serial ports could also be used.

An example of a peripheral device that can be connected to serial ports137 is a Global Positioning System (GPS) 140. It derives 3-dimensionalposition information from a GPS satellite navigation system (not shown).Typically, the 3-dimensional position information is derived by a “GPSCore” module which measures transit times of the L-band signalsbroadcast by the twenty four satellites of the GPS constellation. In thepresent embodiment, GPS 140 is interfaced via one of the serial portsusing NMEA 183 format data. As an alternative, a proprietary format maybe used from one of the many GPS core module makers (e.g., Motorola,Garmin, Trimble, Furuno, Ashtech, Rockwell, Plessy, Canadian Marconi,etc.).

If it is desirable to improve the accuracy of GPS 140, a differentialGPS (DGPS) 141 could be used. It provides correction information to GPSreceiver 140 in order to increase the accuracy and remove the effects ofselective availability. DGPS is developed by a precisely surveyedreference GPS receiver base station. Correction data derived for eachsatellite is formatted in RCTM-104 format and broadcast via acommunications system to the user. These corrections are applied to eachof the measurements made in the users GPS receiver so as to produce amore accurate result. DGPS 141 it is interfaced to the GPS receiver viaa serial interface supporting RCTM-104 format data.

Alternatively, devices that can receive GPS and/or Glonass (GlobalNavigational Satellite System) signals can be connected to serial port137. An example of a device that can receive both GPS and Glonasssignals is GG24 developed by Ashtech Inc. A further alternativeembodiment is to use real time kinematic surveying techniques. Thesetechniques are able to achieve higher accuracy than DGPS.

Another peripheral device that may be used in electro-optic system 100is a spread spectrum (SS) radio 144. It provides wireless communicationbetween peer units or between master and slave units. Major advantagesof SS are spectrum re-use, simultaneous existence of multiple networks,data security and low probability of intercept. An example of a SS radiois Proxim RangeLan 2, which operates at 2.4 GHz and has a data rate of1.6 Mbits per second.

An accelerometer 148 can also be coupled to one of the serial ports. Inone embodiment of accelerometer 148, an integrated circuit (e.g., AnalogDevices' ADXL05) is used to generate an analog voltage which isproportional to the instantaneous acceleration along a specified axis.The analog voltage can be converted to a digital value (by ananalog-to-digital converter) and then serialized so as to be compatiblewith a chosen serial port communication protocol. The accelerationinformation could be used in image stabilization efforts and inaugmenting the information from the GPS and tri-axial magnetometers.

A tri-axial magnetometer, such as a TCM-2 module from PrecisionNavigation Inc., can also be connected to one of the serial ports. Sucha device provides attitude information, in all three degrees of freedom,regarding the pointing direction of the optical axis of the camera.

In FIG. 2, a laser range finder 162 is connected to serial port 160. Anexample of a range finder is Leica's Data Disto RS232.

System 100 also allows various user interface devices 154 to beconnected to ISA bus 126. These interface devices include devices thatcan accept input signal and generate output signals. Examples of userinterface devices are control buttons, switches, optical indicators(e.g., LEDs) and alarms.

In the present specification, four exemplary applications of system 100are described. The first application is an electronic binoculars havinga “text box” superimposed on a real image. The text box contains textdata related to the real image. The second application is “0-0”visibility navigation system which can help a user to navigate a movableobject (e.g., ships, planes, and vehicles) under adverse visualenvironment. The third application is an object identification systemwhich attempts to identify an object under adverse viewing conditions.The fourth application is an advanced image augmentation applicationwhich can process, enhance, and augment images.

Electronic Binocular

Camera 102 is used to capture a view and delivers a corresponding videodata to FireWire chipset 104. The data flows to IP 110. In thisapplication, IP 110 is configured as a pass-through device and justpasses the data, without processing, on to graphics processor 116. CPUsystem 114 is not used to process the image data, so none flows over PCIbus 108. CPU system 114 sends text data to graphics processor 116 whichrenders it at points in the image that correspond to the attitude andlocation “text boxes.” Location data is read from GPS 140 and ifappropriate, DGPS 141, via serial port 137. Attitude information is readfrom tri-axial magnetometer 150 via serial port interface 136. GIS datais retrieved from the database on storage system unit 130.

In one embodiment, CPU system 114 sets the optical and electronic zoomfactors in camera 102. Zoom factors are read back along with exposureand iris information from the camera via the FireWire. Information onzoom factors is used by CPU system 114 to properly generate augmentationimages.

“0-0” Visibility Navigation

Camera 102 is used to capture a view and deliver a corresponding videodata to FireWire chipset 104. The data flows to IP 110, which attemptsto extract features from the data using a plurality of frames. Locationdata is read from GPS 140 and if appropriate, DGPS 141, via serial port137. Attitude information is read from tri-axial magnetometer 150 viaserial port interface 136. CPU system 114 retrieves GIS data fromstorage unit 130 relative to the current position. It then sends “wireframe” graphics to graphics processor 116, which renders and texturesthose wire frames into realistic looking images of what “should” be inthe field of view of the system.

Vessel Identification

Camera 102 is used to capture a view and delivers a corresponding videodata to FireWire chipset 104. The data flows to IP 110. CPU system 114has previously retrieved (at the user's request) several 3D models ofships that are due to pass by. The 3D models of ships are retrieved fromstorage unit 130. This information is sent to IP 110 via PCI bus 108. IP110 searches the field of view (obtained from camera 102) for objects.Upon identifying an object, it is compared with various aspects of the3D models. Upon finding a “match,” information is sent to CPU system114, which sends a signal to alert the user.

Advanced Image Augmentation

Camera 102 is used to capture a view and deliver a corresponding videodata to FireWire chipset 104. The data flows to IP 110, which enhancesthe image and attempts to extract features from the data using aplurality of frames. CPU system 114 has previously loaded IP 110 withthe appropriate algorithm and program information. The enhanced imageflows to graphics processor 116 (via Zoom Video 106) and the features toCPU system 114 (via PCI bus 108). Meanwhile the image data also flows toCPU system 114 (via PCI bus 108) which uses the feature locations toidentify objects in the image and create a “mask” to be used by graphicsprocessor 116 to “remove” those features from the final view. CPU system114 sends the “masks” to graphics processor 116 along with text anddrawing primitives. Graphics processor 116 renders the text, removes the“masked” objects, and renders non-existent objects as directed by CPUsystem 114.

Location data is read from GPS 140 and if appropriate, DGPS 141, viaserial port 137. Attitude information is read from tri-axialmagnetometer 150 via serial port interface 136. CPU system 114 retrievesGIS data relative to the current position from storage unit 130.

In this application, camera 102 is configured to achieve a desiredstabilization factors. Graphics processor 116 is configured to displayimage data (received from Zoom Video bus 106) while applying the “mask”,text and graphics as overlays. Zoom factors are read back along withexposure and iris information from camera 104 via FireWire chipset 104.

The invention now being fully described, it will be apparent to one ofordinary skill in the art that any changes and modifications can be madethereto without departing from the spirit or scope of the invention asset forth herein. It should be noted that computing technology isconstantly being developed. New developments can be appropriately usedto improve the system disclosed herein. For example, a new peripheralbus called the Universal Serial Bus (USB) may be advantageously used toconnect a large number of peripherals to the system of the presentinvention. Similarly, various solid state memory devices such assynchronous DRAM, EDRAM, etc. can also be used. Cache memory can beattached to the present system to improve the performance. Accordingly,the present invention is to be limited solely by the scope of theappended claims.

1. (canceled)
 2. An image processing system comprising: a real timeposition device identifying a position of at least a portion of theimage processing system, wherein the position identifies informationenhancing real scene image information at the position; a graphicprocessor for processing and combining the information enhancing thereal scene image information with the real scene image information toproduce an augmented real scene image for viewing; and a display forshowing the augmented real scene image that is provided in visualproximity of the position identified by the real time position device.3. The system of claim 2, wherein the real time position device includesa global positioning device.
 4. The system of claim 3, wherein the realtime position device includes a Global Positioning System device.
 5. Thesystem of claim 3, wherein the real time position device includes adifferential Global Positioning System device.
 6. The system of claim 3,wherein the real time position device includes a Global NavigationalSatellite System device.
 7. The system of claim 2 further comprising adatabase for storing information enhancing real scene image information.8. An image processing system comprising: an attitude device identifyingan attitude of at least a portion of the image processing system,wherein the attitude identifies information enhancing real scene imageinformation at the device; a graphic processor for processing andcombining the information enhancing the real scene image informationwith the real scene image information to produce an augmented real sceneimage for viewing; and a display showing the augmented real scene imagethat is provided in visual proximity of the attitude identified by theattitude device.
 9. The system of claim 8, wherein the attitude deviceincludes a magnetometer.
 10. The system of claim 8 further comprising adatabase for storing information enhancing real scene image information.